Enhanced image capture

ABSTRACT

Disclosed are techniques that provide a “best” picture taken within a few seconds of the moment when a capture command is received (e.g., when the “shutter” button is pressed). In some situations, several still images are automatically (that is, without the user&#39;s input) captured. These images are compared to find a “best” image that is presented to the photographer for consideration. Video is also captured automatically and analyzed to see if there is an action scene or other motion content around the time of the capture command. If the analysis reveals anything interesting, then the video clip is presented to the photographer. The video clip may be cropped to match the still-capture scene and to remove transitory parts. Higher-precision horizon detection may be provided based on motion analysis and on pixel-data analysis.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. application Ser.No. 17/234,257, filed Apr. 19, 2021, which claims priority to U.S.application Ser. No. 16/289,050, filed Feb. 28, 2019, now U.S. Pat. No.11,019,252, issued May 25, 2021, which claims priority under 35 U.S.C. §120 to U.S. patent application Ser. No. 14/450,492, filed Aug. 4, 2014,now U.S. Pat. No. 10,250,799, issued Apr. 2, 2019, which claims priorityto U.S. Provisional Patent Application 62/001,327, filed on May 21,2014, the disclosures of which are incorporated herein by reference.

The present application is additionally related to U.S. patentapplication Ser. Nos. 14/450,390, 14/450,461, 14/450,522, 14/450,553,and 14/450,573, all filed on Aug. 4, 2014.

TECHNICAL FIELD

The present disclosure is related generally to still-image and videocapture and, more particularly, to digital image processing.

BACKGROUND

On average, people discard a large number of the pictures they take asunsatisfactory. In many cases, this is because the main subject isblinking, moving (i.e., is too blurry), or not smiling at the moment ofimage capture. In other cases, the photographer is inadvertently movingthe image-capture device at the capture moment (e.g., due to an unsteadyhand or to an involuntary rotation of the device). Some pictures arediscarded because the image-capture settings are inappropriate (e.g.,the settings do not accommodate a low-light situation).

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

While the appended claims set forth the features of the presenttechniques with particularity, these techniques, together with theirobjects and advantages, may be best understood from the followingdetailed description taken in conjunction with the accompanying drawingsof which:

FIG. 1A is an overview of a representative environment in which thepresent techniques may be practiced;

FIG. 1B is an overview of a representative network that supports certainof the present techniques;

FIG. 2 is a flowchart of a representative method for selecting andpresenting a “best” captured still image;

FIG. 3 is a flowchart of a representative method for capturing an“interesting” video;

FIG. 4 is a flowchart of a representative method for selecting a “best”captured still image and for capturing an “interesting” video;

FIG. 5 is a flowchart of a representative method for a remote serverthat assists an image-capture device;

FIG. 6 is a flowchart of representative methods for notifying a userthat a “better” still image or an “interesting” video is available;

FIG. 7 is a flowchart of a representative method for detecting a horizonin a captured image and then using the detected horizon; and

FIG. 8 is a schematic showing various components of a representativeimage-capture device or server.

DETAILED DESCRIPTION

Turning to the drawings, wherein like reference numerals refer to likeelements, techniques of the present disclosure are illustrated as beingimplemented in a suitable environment. The following description isbased on embodiments of the claims and should not be taken as limitingthe claims with regard to alternative embodiments that are notexplicitly described herein.

The inventors believe that photographers would like, in addition togetting the best possible photographs, more than one picture to capturethe moment, and, in some cases, a few seconds of video associated with astill picture. This later should be accomplished without thephotographer having to spend the time to switch between still-capturemode and video-capture mode.

Aspects of the presently disclosed techniques provide a “best” picturetaken within a few seconds of the moment when a capture command isreceived (e.g., when the “shutter” button is pressed). Also, severalseconds of video are captured around the same time and are madeavailable to the photographer. More specifically, in some embodiments,several still images are automatically (that is, without the user'sinput) captured. These images are compared to find a “best” image thatis presented to the photographer for consideration. Video is alsocaptured automatically and analyzed to see if there is an action sceneor other motion content around the time of the capture command. If theanalysis reveals anything interesting, then the video clip is presentedto the photographer. The video clip may be cropped to match thestill-capture scene and to remove transitory parts. In furtherembodiments, better low-light images are provided by enhancing exposurecontrol. Higher-precision horizon detection may be provided based onmotion analysis.

For a more detailed analysis, turn first to FIG. 1A. In this exampleenvironment 100, a photographer 102 (also sometimes called the “user” inthis discussion) wields his camera 104 to take a still image of the“scene” 106. In this example, the photographer 102 wants to take asnapshot that captures his friend 108.

The view that the photographer 102 actually sees is depicted as 110,expanded in the bottom half of FIG. 1A. Specifically, when thephotographer 102 pushes a “capture” button (also called the “shutter”for historical reasons), the camera 104 captures an image and displaysthat captured image in the viewfinder display 112. So far, this shouldbe very familiar to anyone who has ever taken a picture with asmartphone or with a camera that has a large viewfinder display 112. Inthe example of FIG. 1A, however, the camera 104 also displays a“notification icon” 114 to the photographer 102. While the detailedfunctioning supporting this icon 114 is discussed at length below, inshort, this icon 114 tells the photographer 102 that the camera 104believes that it has either captured a “better” still image than the onedisplayed in the viewfinder display 112 or that it has captured a videothat may be of interest to the photographer 102.

FIG. 1B introduces a network 116 (e.g., the Internet) and a remoteserver 118. The discussion below shows how these can be used to expandupon the sample situation of FIG. 1A. FIG. 1B also visually makes thepoint that the “camera” 104 need not actually be a dedicated camera: Itcould be any image-capture device including a video camera, a tabletcomputer, smartphone, and the like. For clarity's sake, the presentdiscussion continues to call the image-capture device 104 a “camera.”

FIG. 2 presents methods for specific techniques that enhance still-imagecapture. In step 200, the camera 104 captures a number of still images.Consider, for example, the photographer 102 putting the camera 104 into“viewfinder” mode. In this mode, the camera's viewfinder 112 displaysthe image “seen” by the camera 104. The photographer 102 may explicitlycommand the camera 104 to enter this mode, or the camera 104 canautomatically enter this mode when it determines that this mode isdesired (e.g., by monitoring the camera's current position and observingthe behavior of the photographer 102).

In any case, the camera 104 automatically (that is, while still inviewfinder mode and not in response to an explicit command from thephotographer 102) captures a number of still images, e.g., five persecond over a period of a couple of seconds. These captured still imagesare stored by the camera 104.

In taking so many images, memory storage often becomes an issue. In someembodiments, the images are stored in a circular buffer (optional step202) holding, say, ten seconds of still images. Because the capacity ofthe circular buffer is finite, the buffer may be continuously refreshedwith the latest image replacing the earliest one in the buffer. Thus,the buffer stores a number of captured still images ranging in time fromthe newest image back to the oldest, the number of images in the bufferdepending upon the size of the buffer. In some embodiments, theselection process (see the discussion of step 208 below) is performedcontinuously on the set of images contained in the circular buffer.Images that are not very good (as judged by the techniques discussedbelow) are discarded, further freeing up space in the circular bufferand leaving only the “best” images captured over the past, say, threeseconds. Even in this case, the metadata associated with discardedimages are kept for evaluation.

Note that the capture rate of images in step 200 may be configurable bythe photographer 102 or may depend upon an analysis of thephotographer's previous behavior or even upon an analysis of thecaptured images themselves. If, for example, a comparison of one imageto another indicates a significant amount of movement in the capturedscene, then maybe the camera 104 is focused on a sporting event, and itshould increase its capture rate. The capture rate could also dependupon the resources available to the camera 104. Thus, if the camera'sbattery is running low, then it may reduce the capture rate to conserveenergy. In extreme cases, the technique of automatic capture can beturned off when resources are scarce.

At step 204 (generally while the camera 104 continues to automaticallycapture still images), the photographer 102 gives a capture command tothe camera 104. As mentioned above, this can result from thephotographer 102 pressing a shutter button on the camera 104. (Ingeneral, the capture command can be a command to capture one still imageor a command to capture a video.)

(For purposes of the present discussion, when the camera 104 receivesthe capture command, it exits the viewfinder mode temporarily and entersthe “capture” mode. Once the requested still image (or video asdiscussed below) is captured, the camera 104 generally re-entersviewfinder mode and continues to automatically capture images per step200.)

Unlike in the technique of step 200, traditional cameras stay in theviewfinder mode without capturing images until they receive a capturecommand They then capture the current image and store it. A camera 104acting according to the present techniques, however, is alreadycapturing and storing images (steps 200 and 202) even while it is stillin the viewfinder mode. One way of thinking about the present techniquesis to consider the capture command of step 204 not to be a command atall but rather to be an indication given by the photographer 102 to thecamera 104 that the photographer 102 is interested in something that heis seeing in the viewfinder display 112. The camera 104 then actsaccordingly (that is, it acts according to the remainder of theflowchart of FIG. 2 .)

Step 206 is discussed below in conjunction with the discussion of step214.

In step 208, the camera 104 reviews the images it has captured (whichmay include images captured shortly before or shortly after the capturecommand is received) and selects a “best” one (or a “best” several insome embodiments). (In some embodiments, this selection process isperformed on partially processed, or “raw,” images.) Many differentfactors can be reviewed during this analysis. As mentioned above, thecapture command can be considered to be an indication that thephotographer 102 is interested in what he sees. Thus, a very short timeinterval between the capture command and the time that a particularimage was captured means that that particular image is likely to be ofsomething that the photographer 102 wants to record, and, thus, thistime interval is a factor in determining which image is “best.”

Various embodiments use various sets of information in deciding which ofthe captured images is “best.” In addition to temporal proximity to thephotographer's capture command, some embodiments use motion-sensor data(from an accelerometer, gyroscope, orientation, or GPS receiver on thecamera 104) (e.g., was the camera 104 moving when this image wascaptured?), face-detection information (face detection, position, smileand blink detection) (i.e., easy-to-detect faces often make for goodsnapshots), pixel-frame statistics (e.g., statistics of luminance:gradient mean, image to image difference), activity detection, data fromother sensors on the camera 104, and scene analysis. Furtherinformation, sometimes available, can include a stated preference of thephotographer 102, past behavior of the photographer 102 (e.g., thisphotographer 102 tends to keep pictures with prominent facial images),and a privacy setting (e.g., do not keep pictures with a prominent faceof a person who is not in a list of contacts for the camera 104). Also,often available are camera 104 metadata and camera-status information.All such data can be produced in the camera 104 and stored as metadataassociated with the captured images.

These metadata may also include reduced resolution versions of thecaptured images which can be used for motion detection within thecaptured scene. Motion detection provides information which is used for“best” picture selection (and analysis of captured video, see discussionbelow), as well as other features which improve the image-captureexperience.

The statistics and motion-detection results can also be used by anexposure procedure to improve captured-image quality in low light by,for example, changing exposure parameters and flash lighting. When thereis motion in low light and strobe lighting is available from the camera104, the strobe may be controlled such that multiple images can becaptured with correct exposures and then analyzed to select the bestexposure.

However, the “best” captured image is selected, that best image ispresented to the photographer 102 is step 210. There are severalpossible ways of doing this. Many embodiments are intended to becompletely “transparent” from the photographer's perspective, that is,the photographer 102 simply “snaps” the shutter and is presented withthe selected best image, whether or not that is actually the imagecaptured at the time of the shutter command.

Consider again the situation of FIG. 1A. When the photographer 102presses the shutter button (step 204), the viewfinder display 112 is asshown in FIG. 1A. Clearly, the photographer 102 wants a picture of theface of his friend 108. The system can review the captured images from,say a second before to a second after the capture command is received,analyze them, and then select the best one. Here, that would be an imagethat is in focus, in which the friend 108 is looking at the camera 104,has her eyes open, etc. That best image is presented to the photographer102 when he presses the shutter button even if the image captured at theexact time of the shutter press is not as good.

A slightly more complicated user interface presents the photographer 102with the image captured when the shutter command was received (as istraditional) and then, if that image is not the best available, presentsthe photographer 102 with an indication (114 in FIG. 1A) that a “better”image is available for the photographer's consideration. Again,considering the situation of FIG. 1A, maybe his friend 108 blinks at thetime of the capture command That ‘blinking” image is presented to thephotographer 102, but the indication 114 is lit to show that other,possibly better, images are available for the photographer's review.

Other variations on the user interface are possible. The choice of whichto use in a given situation can be based on settings made by thephotographer 102, on an analysis of the photographer's past behavior(e.g., is he a “snapshot tourist,” or does he act more like anexperienced photographer?), and on analysis of the captured scene.

In optional step 212, the selected image is further processed, ifnecessary, and copied to a more permanent storage area.

In some embodiments, the metadata associated with the captured images(possibly including what the photographer 102 eventually does with theimages) are sent (step 214) to a remote server device (118 of FIG. 1B).The work of the remote server 118 is discussed in greater detail belowwith reference to FIG. 5 , but briefly, the remote server 118 analyzesthe information, potentially from multiple image-capture devices 104,looking for trends and for “best practices.” It then encapsulates whatit has learned and sends recommendations to cameras 104 (step 206). Thecameras 104 are free to use these recommendations when they selectimages in step 208.

FIG. 3 presents other methods for enhancing image-capture, this time forvideo images. The method of FIG. 3 can be performed separately from, orin conjunction with, the methods of FIG. 2 .

In step 300, the camera 104 captures video while the camera 104 is inviewfinder mode (that is, as described above, while the camera 104 hasnot received an explicit command to capture video). As with still-imagecapture, parameters of the video capture can be altered to reflect theresources (e.g., battery, memory storage) available on the camera 104.

In some embodiments, the captured video is, at this point, simply a timesequence of “raw,” unprocessed images. (These raw images can be furtherprocessed as necessary later: See the discussion of step 312 below.) Thestorage issues mentioned above for still images are exacerbated forvideo, so, again, a circular buffer is recommended for storing the videoas it is captured (step 302). The latest video images (also called“frames”) replace the oldest ones so that at any time, the circularbuffer has, for example, the last twenty seconds of captured video.

Optionally, a capture command is received in step 304. As discussedabove, this is not treated as an actual command, but rather as anindication given by the photographer 102 to the camera 104 that thephotographer 102 is interested in something that he is seeing in theviewfinder display 112.

Whether a capture command has been received or not, the captured videois continuously analyzed (step 308) to see if it is “interesting.” Whilethe photographer 102 can indicate his interest by pressing the shutter,other information can be used in addition to (or instead of) that, suchas activity detection, intra-frame and inter-frame motion, and facedetection. For example, a sudden surge of activity combined with aclearly recognizable face may indicate an interesting situation. As withstill-image capture, photographer 102 preferences, past behavior, andprivacy settings can also be used in a machine-learning sense to knowwhat this photographer 102 finds interesting.

If a segment (also called a “clip”) of captured video has been found tobe potentially interesting (e.g., if an “interest score” for a videoclip is above a set threshold), then the photographer 102 is notified ofthis in step 308. The photographer 102 may then review the indicatedvideo clip to see if he too finds it to be of interest. If so, then thevideo clip is further processed as necessary (e.g., by applyingvideo-compression techniques) and copied into longer-term storage (step312).

As a refinement, the limits of the interesting video clip can bedetermined using the same analysis techniques described above along withapplying motion-sensor data. For example, the starting point of the clipcan be set shortly before something interesting begins to occur.

Also, as with the still-image embodiments, metadata can be sent to theremote server 118 (step 314). Recommendations and refined operationalparameters, based on analysis performed by the remote server 118, can bereceived (step 306) and used in the analysis of step 308.

Note that from the description above, in some embodiments and in somesituations, the camera 104 captures and presents video without everleaving the viewfinder mode. That is, the camera 104 views the scene,delimits video clips of interest, and notifies the photographer 102 ofthese video clips without ever receiving any explicit command to do so.In other embodiments, these video-capture and analysis techniques can beexplicitly invoked or disabled by the photographer 102.

As mentioned above in the introduction to the discussion of FIG. 3 , thestill-image capture-enhancement techniques of FIG. 2 can be combinedwith the video-image capture-enhancement techniques of FIG. 3 . FIG. 4presents such a combination with some interesting refinements.

Consider once again the scenario of FIG. 1A. The camera 104 is inviewfinder mode, capturing both still images (step 400, as per step 200of FIG. 2 ) and video (step 408, as in step 300 of FIG. 3 ). In theproper circumstances, the system presents both the best captured stillimage (step 406) and interesting video (step 410) for the photographer'sconsideration (possibly using the time of the capture command of step402 to select and analyze the captured images and frames).

Even though still images and video frames can be captured at the sametime, the refinement of FIG. 4 applies image-stabilization techniques tothe captured video but not to the captured still images (step 412). Thisprovides both better video and better stills than would any known“compromise” system that does the same processing for both stills andvideo.

In another refinement, the selection of the best still image (step 406)can depend, in part, on the analysis of the video (step 410) and viceversa. Consider a high-motion sports scene. The most important scenesmay be best determined from analyzing the video because that will bestshow the action. From this, the time of the most interesting moment isdetermined. That determination may alter the selection process of thebest still image. Thus, a still image taken at the moment when a playerkicks the winning goal may be selected as the best image, even thoughother factors may have to be compromised (e.g. the player's face is notclearly visible in that image). Going in the other direction, a videoclip may be determined to be interesting simply because it contains anexcellent view of a person's face even though that person is not doinganything extraordinary during the video.

Specifically, all of the metadata used in still-image selection can beused in combination with all of the metadata used in video analysis anddelimitation. The combined metadata set can then be used to both selectthe best still image and to determine whether or not a video clip isinteresting.

The methods of FIG. 4 can also include refinements in the use of theremote server 118 (steps 404 and 414). These refinements are discussedbelow in reference to FIG. 5 .

Methods of operation of the remote server 118 are illustrated in FIG. 5. As discussed above, the server 118 receives metadata associated withstill-image selection (step 500; see also step 214 of FIG. 2 and step414 of FIG. 4 ). The same server 118 may also receive metadataassociated with analyzing videos to see if they are interesting (step504; see also step 314 of FIG. 3 and step 414 of FIG. 4 ). The server118 can analyze these two data sets separately (step 508) and providestill-image selection recommendations (step 510) and video-analysisrecommendations (step 510) to various image-capture devices 104.

In some embodiments, however, the remote server 118 can do more. First,in addition to analyzing metadata, it can further analyze the datathemselves (that is, the actual captured still images and video) if thatcontent is made available to it by the image-capture devices 104 (steps502 and 506). With the metadata and the captured content, the server 118can perform the same kind of selection and analysis performed locally bythe image-capture devices 104 themselves (see step 208 of FIG. 2 ; steps308 and 310 of FIG. 3 ; and steps 406 and 410 of FIG. 4 ). Rather thansimply providing a means for second-guessing the local devices 104, theserver 118 can compare its own selections and interest scores againstthose locally generated and thus refine its own techniques to bettermatch those in the general population of image-capture devices 104.

Further, the image-capture device 104 can tell the remote server 118just what the photographer 102 did with the selected still images andthe video clips thought to be interesting (steps 502 and 506). Again,the server 118 can use this to further improve its recommendationmodels. If, for example, photographers 102 very often discard thosestill images selected as best by the techniques described above, then itis clear that those techniques may need to be improved. The server 118may be able to compare an image actually kept by the photographer 102against the image selected by the system and, by analyzing over a largepopulation set, learn better how to select the “best” image.

Going still further, the remote server 118 can analyze thestill-image-selection metadata (and, if available, the still imagesthemselves and the photographer's ultimate disposition of the stillimages) together with the video-analysis metadata (and, if available,the video clips themselves and the photographer's ultimate dispositionof the captured video). This is similar to the cross-pollination conceptdiscussed above with respect to FIG. 4 : That is, by combining theanalysis of still images and video, the server 118 can further improveits recommendations for both selecting still images and for analyzingvideo clips. The particular methodologies usable here are well knownfrom the arts of pattern analysis and machine learning.

In sum, if the remote server 118 is given access to information aboutthe selections and analyses of multiple image-capture devices 104, thenfrom working with that information, the server 118 can provide betterrecommendations, either generically or tailored to particularphotographers 102 and situations.

FIG. 6 presents methods for a user interface applicable to the presentlydiscussed techniques. Much of the user-interface functionality hasalready been discussed above, so only a few points are discussed in anydetail here.

In step 600, the camera 104 optionally enters the viewfinder modewherein the camera 104 displays what it sees in the viewfinder display112. As mentioned above with reference to FIG. 2 , the photographer 102may explicitly command the camera 104 to enter this mode, or the camera104 can automatically enter this mode when it determines that this modeis desired.

In a first embodiment of step 602, the photographer 102 presses theshutter button (that is, submits an image-capture command to the camera104), the camera 104 momentarily enters the image-capture mode, displaysa captured image in the viewfinder display 112, and then re-entersviewfinder mode. In a second embodiment, the photographer puts thecamera 104 into another mode (e.g., a “gallery” mode) where it displaysalready captured images, including images automatically captured.

As discussed above, the displayed image can either be one captureddirectly in response to an image-capture command or could be a “better”image as selected by the techniques discussed above. If there is acaptured image that is better than the one displayed, then thephotographer 102 is notified of this (step 604). The notification can bevisual (e.g., by the icon 114 of FIG. 1A), aural, or even haptic. Insome cases, the notification is a small version of the better imageitself. If the photographer 102 clicks on the small version, then thefull image is presented in the viewfinder display 112 for hisconsideration. While the camera 104 is in gallery mode, the photographer102 can be notified of which images are “better” by highlighting them insome way, for example by surrounding them with a distinctive border orshowing them first.

Meanwhile, a different user notification can be posted if the techniquesabove capture a video clip deemed to be interesting. Again, severaltypes of notification are possible, including a small still from thevideo (or even a presentation of the video itself).

Other user interfaces are possible. While the techniques described abovefor selecting a still image and for analyzing a video clip are quitesophisticated, they allow for a very simple user interface, in somecases an interface completely transparent to the photographer 102 (e.g.,just show the best captured still image when the photographer 102presses the shutter button). More sophisticated user interfaces areappropriate for more sophisticated photographers 102.

FIG. 7 presents a refinement that can be used with any of the techniquesdescribed above. A first image (a still or a frame of a video) iscaptured in step 700. Optionally, additional images are captured in step702.

In step 704, the first image is analyzed (e.g., looking for horizontalor vertical lines). Also, motion-sensor data from the camera 104 areanalyzed to try to determine the horizon in the first image.

Once the horizon has been detected, it can be used as input whenselecting other images captured close in time to the first image. Forexample, the detected horizon can tell how level the camera 104 was heldwhen an image was captured, and that can be a factor in determiningwhether that image is better than another. Also, the detected horizoncan be used when post-processing images to rotate them into level or tootherwise adjust them for involuntary rotation.

FIG. 8 shows the major components of a representative camera 104 orserver 118. The camera 104 could be, for example, a smartphone, tablet,personal computer, electronic book, or dedicated camera. The server 118could be a personal computer, a compute server, or a coordinated groupof compute servers.

The central processing unit (“CPU”) 800 of the camera 104 or server 118includes one or more processors (i.e., any of microprocessors,controllers, and the like) or a processor and memory system whichprocesses computer-executable instructions to control the operation ofthe device 104, 118. In particular, the CPU 800 supports aspects of thepresent disclosure as illustrated in FIGS. 1 through 7 , discussedabove. The device 104, 118 can be implemented with a combination ofsoftware, hardware, firmware, and fixed-logic circuitry implemented inconnection with processing and control circuits, generally identified at802. Although not shown, the device 104, 118 can include a system bus ordata-transfer system that couples the various components within thedevice 104, 118. A system bus can include any combination of differentbus structures, such as a memory bus or memory controller, a peripheralbus, a universal serial bus, and a processor or local bus that utilizesany of a variety of bus architectures.

The camera 104 or server 118 also includes one or more memory devices804 that enable data storage (including the circular buffers describedin reference to FIGS. 2 through 4 ), examples of which includerandom-access memory, non-volatile memory (e.g., read-only memory, flashmemory, erasable programmable read-only memory, and electricallyerasable programmable read-only memory), and a disk storage device. Adisk storage device may be implemented as any type of magnetic oroptical storage device, such as a hard disk drive, a recordable orrewriteable disc, any type of a digital versatile disc, and the like.The device 104, 118 may also include a mass-storage media device.

The memory system 804 provides data-storage mechanisms to store devicedata 812, other types of information and data, and various deviceapplications 810. An operating system 806 can be maintained as softwareinstructions within the memory 804 and executed by the CPU 800. Thedevice applications 810 may also include a device manager, such as anyform of a control application or software application. The utilities 808may include a signal-processing and control module, code that is nativeto a particular component of the camera 104 or server 118, ahardware-abstraction layer for a particular component, and so on.

The camera 104 or server 118 can also include an audio-processing system814 that processes audio data and controls an audio system 816 (whichmay include, for example, speakers). A visual-processing system 818processes graphics commands and visual data and controls a displaysystem 820 that can include, for example, a display screen 112. Theaudio system 816 and the display system 820 may include any devices thatprocess, display, or otherwise render audio, video, display, or imagedata. Display data and audio signals can be communicated to an audiocomponent or to a display component via a radio-frequency link, S-videolink, High-Definition Multimedia Interface, composite-video link,component-video link, Digital Video Interface, analog audio connection,or other similar communication link, represented by the media-data ports822. In some implementations, the audio system 816 and the displaysystem 820 are components external to the device 104, 118. Alternatively(e.g., in a cellular telephone), these systems 816, 820 are integratedcomponents of the device 104, 118.

The camera 104 or server 118 can include a communications interfacewhich includes communication transceivers 824 that enable wired orwireless communication. Example transceivers 824 include WirelessPersonal Area Network radios compliant with various Institute ofElectrical and Electronics Engineers (“IEEE”) 802.15 standards, WirelessLocal Area Network radios compliant with any of the various IEEE 802.11standards, Wireless Wide Area Network cellular radios compliant with 3rdGeneration Partnership Project standards, Wireless Metropolitan AreaNetwork radios compliant with various IEEE 802.16 standards, and wiredLocal Area Network Ethernet transceivers.

The camera 104 or server 118 may also include one or more data-inputports 826 via which any type of data, media content, or inputs can bereceived, such as user-selectable inputs (e.g., from a keyboard, from atouch-sensitive input screen, or from another user-input device),messages, music, television content, recorded video content, and anyother type of audio, video, or image data received from any content ordata source. The data-input ports 826 may include Universal Serial Busports, coaxial-cable ports, and other serial or parallel connectors(including internal connectors) for flash memory, storage disks, and thelike. These data-input ports 826 may be used to couple the device 104,118 to components, peripherals, or accessories such as microphones andcameras.

Finally, the camera 104 or server 118 may include any number of “othersensors” 828. These sensors 828 can include, for example,accelerometers, a GPS receiver, compass, magnetic-field sensor, and thelike.

The remainder of this discussion presents details of choices andprocedures that can be used in certain implementations. Although quitespecific, these details are given so that the reader can more fullyunderstand the broad concepts discussed above. These implementationchoices are not intended to limit the scope of the claimed invention inany way.

Many techniques can be used to evaluate still images in order to selectthe “best” one (step 208 of FIG. 2 ). For images that contain faces, oneembodiment calculates an image score based on sharpness and exposure andcalculates a separate score for facial features.

First, facial-recognition techniques are applied to the captured imagesto see if many of them contain faces. If so, then the scene beingcaptured is evaluated as a “face” scene. If the scene is not a facescene, then the sharpness/exposure score is used by itself to select thebest image. For a face scene, on the other hand, if the images availablefor evaluation (that is, the set of all captured images that arereasonably close in time to the capture command) have very similarsharpness/exposure scores (e.g., the scores are equal within asimilarity threshold which can be specific to the hardware used), thenthe best image is selected based purely on the face score.

For a face scene when the set of images have significant differences intheir sharpness/exposure scores, then the best image is the one that hasthe highest combination score based on both the sharpness/exposure scoreand the face score. The combination score may be a sum or weighted sumof the two scores:picture e _(score)(i)=mFEscore(i)+total_(faces)(i)

The sharpness/exposure score can be calculated using the mean of theSobel gradient measure for all pixels in the image and the mean pixeldifference between the image being analyzed and the immediatelypreceding image. Luminance-only data are used in these calculations. Theframe-gradient metric and frame-difference metric are calculated as:

${mSobel} = {\frac{1}{WH}{\sum\limits_{i = 1}^{W}{\sum\limits_{j = 1}^{H}{0.5*\left( {{{{abs}\left( {{Sobel\_ x}\left( {i,j} \right)} \right)} + {{{abs}\left( {{Sobel\_ y}\left( {i,j} \right)} \right)}{mDiff}}} = {\frac{1}{WH}{\sum\limits_{i = 1}^{W}{\sum\limits_{j = 1}^{H}{{abs}\left( {{Y_{t}\left( {i,j,t} \right)} - {Y_{t - 1}\left( {i,j} \right)}} \right)}}}}} \right.}}}}$where:

W=image width;

H=image height;

Sobel_x=The result of convolution of the image with the Sobel Gxoperator:

$G_{x} = \begin{bmatrix}{- 1} & 0 & 1 \\{- 2} & 0 & 2 \\{- 1} & 0 & 1\end{bmatrix}$

Sobel_y=The result of convolution of the image with the Sobel Gyoperator:

$G_{y} = \begin{bmatrix}1 & 2 & 1 \\0 & 0 & 0 \\{- 1} & {- 2} & {- 1}\end{bmatrix}$

The sharpness/exposure score is calculated for each image (i) in thecircular image buffer of N images around the capture moment using theSobel value and its minimum:

${{mFEscore}(i)} = {\left( {{{mSobel}(i)} - {\min\limits_{N}({mSobel})} + 1} \right)*\left( {1 - \frac{{mDiff}(i)}{200}} \right)}$

The mFEscore is set to 0 for any image if the mean of all pixel valuesin the image is not within a normal exposure range or if the focus stateindicates that the image is out-of-focus. The sharpness/exposure scorevalues for the set of available images are then normalized to a rangeof, say, 0 to 100 to be used in conjunction with face scores, when aface scene is detected.

The face score is calculated for the images when at least one face isdetected. For each face, the score consists of a weighted sum ofdetected-smile score, open-eyes score, and face-orientation score. Forexample:

-   -   Smile: Values range from 1 to 100 with large values for a wide        smile, small values for no smile.    -   Eyes Open: Values range from 1 to 100 with small values for        wide-open eyes, large values for closed eyes (e.g., a blink).        Values are provided for each eye separately.    -   A separate blink detector may also be used.    -   Face Orientation (Gaze): An angle from 0 for a frontal look to        +/−45 for a sideways look.

The procedure uses face-detection-engine values and creates normalizedscores for each of the face parameters as follows:

-   -   Smile Score: Use the smile value from the engine; then normalize        to a 1 to 100 range for the set of N available images as        follows:

${{smile}(i)} = \frac{{{smile}(i)} - {\min\limits_{N}({smile})}}{{\max\limits_{N}({smile})} - {\min\limits_{N}({smile})}}$

-   -   Eyes-Open Score: Detect the presence of a blink or half-opened        eyes using the blink detector and a change-of-eyes parameters        between consecutive frames; score O for images when a blink or        half-open eye is detected. For the rest of the images, a score        is calculated using the average of the values for both eyes and        normalizing to the range in a manner similar to that described        for a smile. The maximum score is obtained when the eyes are        widest open over the N images in the analysis.    -   Face-Orientation Score (Gaze): Use a maximum score for a frontal        gaze and reduce the score when the face is looking sideways        For each face in the image, a face score is calculated as a        weighted sum:        face_(score)=α*smile+β*eyes+π*gaze

If there are more faces than one in an image, then an average orweighted average of all face scores can be used to calculate the totalface score for that image. The weights used to calculate total facescore could correlate to the face size, such that larger faces havehigher score contributions to the total face score. In anotherembodiment, weights correlate with face priority determined throughposition or by some face-recognition engine. For an image (i) with Mfaces, the total faces score then may be calculated as:

${{tota}{l_{faces}(i)}} = \frac{\sum\limits_{j = 1}^{M}{w_{j}*{fac}{e_{score}(j)}}}{\sum\limits_{j = 1}^{M}w_{j}}$

As discussed above, the face score can then be combined (as appropriate)with the sharpness/exposure score, and the image with the highest scoreis selected as the “best” image. As a refinement, in some embodiments,the selected image is then compared against the “captured” image (thatis, the image captured closest in time to the time of the capturecommand). If these images are too similar, then only the captured imageis presented to the user. This consideration is generally applicablebecause studies have shown that photographers do not prefer the selected“best” image when its differences from the captured image are quitesmall.

As with selecting a “best” image, many techniques can be applied todetermine whether or not a captured video is “interesting.” Generally,the video-analysis procedure runs in real time, constantly marking videoframes as interesting or not. Also, the video analysis determines wherethe interesting video clip begins and ends. Some metrics useful in videoanalysis include region of interest, motion vectors (“MVs”), devicemotion, face information, and frame statistics. These metrics arecalculated per frame and associated with the frame.

In some embodiments, a device-motion detection procedure combines datafrom a gyroscope, accelerometer, and magnetometer to calculate devicemovement and device position, possibly using a complementary filter orKalman filter. The results are categorized as follows:

-   -   NO_MOTION means that the device is either not moving or is        experiencing only a small level of handshake;    -   INTENTIONAL_MOTION means that the device has been intentional        moved (e.g., the photographer is panning); and    -   UNINTENTIONAL_MOTION means that the device has experienced large        motion that was not intended as input to the image-capture        system (e.g., the device was dropped, pulled out of a pocket,        etc.).        By comparing consecutive values of the calculated position, the        device's motion in three spatial axes is characterized:    -   if (delta position of all 3-axis<NO_MOVEMENT_THRESHOLD)        -   device motion state=NO_MOTION    -   if (delta position of one axis<INTENTIONAL_MOTION_THRESHOLD &&        delta position of other two axis<NO_MOVEMENT_THRESHOLD && occurs        over a sequence of frames)        -   device motion state=INTENTIONAL_MOTION    -   if (delta position of any axis>UNINTENTIONAL_MOTION_THRESHOLD)        -   device motion state=UNINTENTIONAL_MOTION            The device-motion state is then stored in association with            the image.

Motion estimation finds movement within a frame (intra-frame) as opposedto finding movement between frames (inter-frame). A block-basedmotion-estimation scheme uses a sum of absolute differences (“SAD”) asthe primary cost metric. Other embodiments may use object tracking.Generic motion-estimation equations include:

${{SA{D\left( {i,j} \right)}} = {\sum\limits_{x = 0}^{N - 1}{\sum\limits_{y = 0}^{N - 1}{❘{{s\left( {x,y,l} \right)} - {s\left( {{x + i},{y + j},{k - l}} \right)}}❘}}}}{{{{s\left( {x,y,l} \right)}{where}0} \leq x},{y \leq {N - 1}}}{\left\lbrack {{vx},{vy}} \right\rbrack = {\arg{\min\limits_{i,j}\left\lbrack {SA{D\left\lbrack {i,j} \right)}} \right\rbrack}}}$where:

S(x, y, l) is a function specifying pixel location;

(l)=candidate frame;

-   -   (k)=reference frame; and

(vx, vy) is the motion-vector displacement with respect to (i, j).

The motion-estimation procedure compares each N×N candidate blockagainst a reference frame in the past and calculates the pixeldisplacement of the candidate block. At each displacement position, SADis calculated. The position that produces the minimum SAD valuerepresents the position with the lowest distortion (based on the SADcost metric).

Once the raw vectors are calculated for each N×N block, the vectors arefiltered to obtain the intra-frame motion. In one exemplary method:

-   -   Motion is estimated with predicted motion vectors;    -   The median filter is applied to the motion vectors;    -   Motion vectors are additionally filtered for the following        reasons:        -   IIMVII>a static-motion threshold; or        -   IIMVII>a dynamic-motion threshold; or        -   Collocated zero SAD>mean zero SAD (of all blocks); or        -   Block SAD<a large-SAD threshold; or        -   Luma variance>a low-block-activity threshold;    -   Create a mask region (e.g., inscribe a maximal regular diamond        in the rectangular frame and then inscribe a maximal regular        rectangular (the “inner rectangle”) in the diamond); and    -   Calculate:        -   Diamond_Count=num(MV in the diamond region))/num(MV in the            frame); and        -   Inner_Rectangle_Count=num(MV in the inner rectangle))/num(MV            in the diamond region).

Each frame of video is characterized as “interesting” (or not) based onmetrics such as internal movement in the frame, luma-exposure values,device motion, Sobel-gradient scores, and face motion. These metrics areweighted to account for the priority of each metric.

-   -   Internal Frame Motion: Calculated from Diamond Count and        Inner_Rectangle_Count ratios;    -   Luma Exposure: Calculated from pixel data and weighted less for        over or under exposed images;    -   Sobel-Gradient Scores: Calculated from pixel data and weighted        less for Sobel scores that are far from the temporal average of        Sobel scores for each frame;    -   Device Motion: Uses device-motion states and weighted less for        UNINTENTIONAL_MOTION    -   Face Motion: Motion vectors are calculated from detected        positions for each face. Weighted less for larger motion vectors        for each face.        Putting these together:

${{motion\_ frame}{\_ score}} = {\sum\limits_{i = 0}^{N}{{w(i)}*{metric}(i)}}$

If the motion frame score exceeds a threshold, then the frame isincluded in a “sequence calculation.” This sequence calculation sums upthe number of frames that have interesting information and compares thatto a sequence-score threshold. If the sequence-score is greater than thethreshold, then the scene is marked as an interesting video clip and ispermanently stored (step 312 of FIG. 3 ).

Before a video clip is stored, the start and stop points are calculated.Based on device motion, the first level of delimiters are applied. Theprocedure finds the segment in the video where the device was marked asNO_MOTION and marks the start and stop points. As a secondary check, theprocedure also examines intra-frame motion in each frame and marks thosesub-segments within the segment that have no camera motion to indicatewhen interesting motion occurred in the video. The first frame withinteresting intra-frame motion is the new start of the video clip, andthe last frame after capture in the video with interesting motion endsthe video clip. In some embodiments, the clip is extended to capture asmall amount of time before and after the interesting section.

Horizon detection (see FIG. 7 and accompanying text) processes imageframes and sensor data to find the frame with the most level horizon. Ifnone of the images contain a 0 degree (within a threshold) horizon line,then the image is rotated and cropped to create an image with a levelhorizon. Vertical lines can be used in detecting the horizon as well.

In some embodiments, the following procedure is performed continuously,as each frame arrives. For each image:

-   -   Associate an angle position from the motion sensors with the        image;    -   Apply a Gaussian blur filter followed by an edge-detection        filter on the image (e.g., use a Canny detection filter);    -   Apply image processing to find lines in the image (e.g., use a        Hough Line Transform). For each line found:        -   Calculate the angle of the line with reference to 0 degrees            orientation of the device (i.e., horizontal); and        -   Keep lines that are:            -   Within some angle threshold; and            -   Within some length threshold;    -   Find the longest line (called the “maximal line”), the start and        end positions of the maximal line, and the angle of the maximal        line. (It is useful to store line information in polar and        Cartesian coordinates and in a linear equation.)        At this point in the procedure, each image contains metadata        corresponding to the following parameters: the length of the        maximal line, the maximal line's angle with respect to the        horizon, the linear equation of the maximal line, and the device        orientation (i.e., the angle of the device with respect to the        image plane derived from the motion sensors).

For each series of images, remove from consideration those images wherethe absolute difference (device orientation angle minus angle of themaximal line) is greater than a threshold. This allows physicalmotion-sensor information to be used in conjunction with pixelinformation to determine the angle.

Find the “region of interest” for each image. To do this, extend themaximal line in the image to two boundaries of the image. The region ofinterest is the smallest rectangle bounded on the right and left sidesof the image that contains the maximal line.

Next find the “reference region” by finding the area of greatest overlapamong the regions of interest of the relevant images. This helps verifythat each maximal line is actually the same horizon line but captured atdifferent angles in the different images. Remove from consideration anyimages whose maximal lines fall outside of the reference region.

Finally, for the relevant images, select that image whose maximal linein the reference region has an angle closest to 0 degree orientation(that is, closest to horizontal). Use that as the detected horizon. Ifnecessary, that is, if the angle of the selected image is greater thansome threshold, the rotate the image using the calculated angle and cropand upscale the image.

In view of the many possible embodiments to which the principles of thepresent discussion may be applied, it should be recognized that theembodiments described herein with respect to the drawing figures aremeant to be illustrative only and should not be taken as limiting thescope of the claims. Therefore, the techniques as described hereincontemplate all such embodiments as may come within the scope of thefollowing claims and equivalents thereof.

We claim:
 1. A method comprising: capturing, simultaneously by animage-capture device, a video segment and a sequence of still images;determining, by the image-capture device, that a first score of aquality characteristic for a first still image from the sequence ofstill images is greater than a second score of the qualitycharacteristic for a second still image from the sequence of stillimages; and adjusting, by the image-capture device, the video segmentbased on one or more characteristics of the first still image.
 2. Themethod of claim 1, wherein capturing the video segment and the sequenceof still images is in temporal proximity to a capture command receivedby the image-capture device.
 3. The method of claim 1, furthercomprising presenting, by the image-capture device, the adjusted videosegment and the first still image, and wherein presenting the adjustedvideo segment and the first still image includes presenting the firststill image using a gallery mode.
 4. The method of claim 1, wherein thevideo segment is a first portion of the video segment and the sequenceof still images is a first portion of the sequence of still images, themethod further comprising: adjusting one or more image-capture settingsof the image-capture device to enable a second portion of the videosegment and a second portion of the sequence of still images to becaptured with the adjusted one or more image-capture settings.
 5. Themethod of claim 4, wherein the adjusted one or more image-capturesettings comprises at least one of: exposure parameters of theimage-capture device; or flash lighting of the image-capture device. 6.The method of claim 1, wherein the one or more characteristics of thefirst still image includes a detected horizon.
 7. The method of claim 6,wherein adjusting the video segment comprises: determining a maximalline of one or more frames of the video segment; determining adifference in angle between the maximal line of the one or more framesof the video segment and the detected horizon; and rotating the one ormore frames of the video segment based on the difference in anglebetween the maximal line of the one or more frames of the video segmentand the detected horizon.
 8. The method of claim 7, wherein adjustingthe video segment further comprises determining that the maximal line ofthe one or more frames is indicative of the detected horizon.
 9. Themethod of claim 7, wherein adjusting the video segment further comprisescropping and upscaling the one or more frames of the video segment. 10.The method of claim 1, wherein the sequence of still images correspondsto frames of the video segment.
 11. The method of claim 1, whereindetermining that the first score of the quality characteristic for thefirst still image is greater than the second score of the qualitycharacteristic for the second still image from the sequence of stillimages is based on a machine-learned model, the machine-learned modelconfigured to utilize one or more user preferences, past behaviors, orprivacy settings to determine that the first score of the qualitycharacteristic for the first still image is greater than the secondscore of the quality characteristic for the second still image from thesequence of still images.
 12. The method of claim 1, wherein the firststill image from the sequence of still images and the second still imagefrom the sequence of still images are not consecutive still images inthe sequence of still images.
 13. The method of claim 1, wherein the oneor more characteristics of the first still image do not include aquality characteristic.
 14. An image-capture device comprising: adisplay; at least one processor; and one or more non-transitorycomputer-readable storage media that, when executed by the at least oneprocessor, cause the image-capture device to: capture, simultaneously, avideo segment and a sequence of still images; determine that a firstscore of a quality characteristic for a first still image from thesequence of still images is greater than a second score of the qualitycharacteristic for a second still image from the sequence of stillimages; and adjust the video segment based on one or morecharacteristics of the first still image.
 15. The image-capture deviceof claim 14, wherein the image-capture device is a cellphone, a camera,or a tablet.
 16. The image-capture device of claim 14, wherein the oneor more non-transitory computer-readable storage media, when executed bythe at least one processor, further cause the image-capture device topresent the adjusted video segment and the first still image.
 17. Theimage-capture device of claim 16, wherein the one or more non-transitorycomputer-readable storage media, when executed by the at least oneprocessor, further cause the image-capture device to present theadjusted video segment and the first still image using a gallery mode.18. The image-capture device of claim 14, wherein: the video segment isa first portion of the video segment and the sequence of still images isa first portion of the sequence of still images; and the one or morenon-transitory computer-readable storage media, when executed by the atleast one processor, further cause the image-capture device to adjustone or more image-capture settings of the image-capture device to enablea second portion of the video segment and a second portion of thesequence of still images to be captured with the one or moreimage-capture settings.
 19. The image-capture device of claim 14,wherein the one or more characteristics of the first still imageincludes a detected horizon.
 20. The image-capture device of claim 19,wherein the one or more non-transitory computer-readable storage media,when executed by the at least one processor, further cause theimage-capture device to adjust the video segment by: determining amaximal line of one or more frames of the video segment; determining adifference in angle between the maximal line of the one or more framesof the video segment and the detected horizon; and rotating the one ormore frames of the video segment based on the difference in anglebetween the maximal line of the one or more frames of the video segmentand the detected horizon.
 21. The image-capture device of claim 20, theone or more non-transitory computer-readable storage media, whenexecuted by the at least one processor, further cause the image-capturedevice to adjust the video segment by determining that the maximal lineof the one or more frames are indicative of the detected horizon. 22.The image-capture device of claim 20, wherein the one or morenon-transitory computer-readable storage media, when executed by the atleast one processor, further cause the image-capture device to adjustthe video segment by cropping and upscaling the one or more frames ofthe video segment.